The Internet is a wide area network having a truly global reach, interconnecting computers all over the world. That portion of the Internet generally known as the World Wide Web is a collection of inter-related data whose magnitude is truly staggering. The content of the World Wide Web (sometimes referred to as “the Web”) includes, among other things, documents of the known HTML (Hyper-Text Mark-up Language) format which are transported through the Internet according to the known protocol, HTTP (Hyper-Text Transport Protocol).
The breadth and depth of the content of the Web is amazing and overwhelming to anyone hoping to find specific information therein. Accordingly, an extremely important component of the Web is a search engine. As used herein, a search engine is an interactive system for locating content relevant to one or more user-specified search terms, which collectively represent a search query. Through the known Common Gateway Interface (CGI), the Web can include content which is interactive, i.e., which is responsive to data specified by a human user of a computer connected to the Web. A search engine receives a search query of one or more search terms from the user and presents to the user a list of one or more documents which are determined to be relevant to the search query.
Search engines dramatically improve the efficiency with which users can locate desired information on the Web. As a result, search engines are one of the most commonly used resources of the Internet. An effective search engine can help a user locate very specific information within the billions of documents currently represented within the Web. The critical function and raison d'être of search engines is to identify the few most relevant results among the billions of available documents given a few search terms of a user's query and to do so in as little time as possible. Thus, a critical function of search engines is determination of relevance of documents to a search query.
Generally, search engines maintain a database of records associating search terms with information resources on the Web. Search engines currently acquire information about the contents of the Web primarily in several common ways. The most common is generally known as crawling the Web and the second is by submission of such information by a provider of such information or by third-parties (i.e., neither a provider of the information nor the provider of the search engine). Another common way for search engines to acquire information about the content of the Web is for human editors to create indices of information based on their review.
To understand crawling, one must first understand that documents of the Web can include references, commonly referred to as links, to other documents of the Web. Anyone who has “clicked on” a portion of a document to cause display of a referenced document has activated such a link. Crawling the Web generally refers to an automated process by which documents referenced by one document are retrieved and analyzed and documents referred to by those documents are retrieved and analyzed and the retrieval and analysis are repeated recursively. Thus, an attempt is made to automatically traverse the entirety of the Web to catalog the entirety of the contents of the Web.
Due to the fact that documents of the Web are constantly being added and/or modified and also to the sheer immensity of the Web, no Web crawler has successfully cataloged the entirety of the Web. Accordingly, providers of Web content who wish to have their content included in search engine databases directly submit their content to providers of search engines. Other providers of content and/or services available through the Internet contract with operators of search engines to have their content regularly crawled and updated such that search results include current information. Some search engines, such as the search engine provided by Overture Services, Inc. of Pasadena, Calif. (http://www.overture.com) and described in U.S. Pat. No. 6,269,361 which is incorporated herein by reference, allow providers of Internet content and/or services to compose and submit brief titles and descriptions to be associated with their content and/or services in results as a search query. As the Internet has grown and commercial activity has also grown over the Internet, some search engines have specialized in providing commercial search results presented separately from informational results with the added benefit of facilitating commercial transactions over the Internet. One such search engine is the search engine described in the '361 Patent and provided by Overture Services, Inc. as described above.
Since search engines which provide unwanted information are at a distinct disadvantage to search engines which minimize presentation of unwanted information, search engine providers have a strong interest in maximizing relevance of results provided to search queries. Providers of search engines therefore often review the content of individual search listings for desirability and appropriateness prior to including each listing in their database for real-time delivery of search results in response to a search query.
Due to the overwhelming amount of information on the Web, such review is a daunting task. In addition, content review generally has not lent itself to automation since the appropriateness of a particular search listing depends upon subtleties of human perception of both the search listing itself and the content referenced by the search listing. Operators of search engines have general had to choose between (i) automatically generating search results of listings having questionable relevance and therefore less value to the user or (ii) manually generating more relevant search listings by human editing but on a drastically reduced scale. While manually edited search listings tend to be far more relevant and therefore far more effective in attracting users to a search engine, manual editing of search listings is very expensive in both time and resources and significantly delays availability of newly submitted search listings to users of the search engine. Delayed availability of search listings reduces the currency of search listings produced as results in response to search queries.
What is needed is a mechanism by which review of one or more search listings can be efficiently performed while maintaining accurate analysis of the impression of a given search listing on a human user seeing the search listing and/or the content referenced by the search listing.